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What was asked for 


• From the original ToR: 

- Initially background research and literature reviews will be 
performed as appropriate to inform the framework development. It is 
expected that this will involve identification of best practices and 
lessons learned from the Air Force and other organizations which 
have implemented enterprise risk management processes, as well 
as internal best practices. 

- Next, there are two approaches that will be developed in Phase 1 to 
identify and assess strategic risks. The first approach is a “bottom- 
up” approach, where risks are collected from existing sources 
including, but not limited to: budgetary issues across the agency, 
OCE risk issues, and BPR State of the Agency “Red-Yellow-Green” 
data collection. The second approach is to identify agency-level 
risks that do not result from the first approach. This is a “top-down” 
approach, where these risks would represent those of an 
institutional and/or non-project/program nature, and may involve risk 
identification with subject matter experts from PA&E and with the 
core team members. 

- Finally, a methodology for normalizing and ranking identified 
strategic risks against each other will be developed. 
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Approach 


• Hired the Aerospace Corporation to assist 

- Work with Strategic Missile Command in risk ranking & tracking 

• Performed a literature search on risk management 

- Both internal to NASA and external 

• Developed two potential risk frameworks 

- Based on Aerospace experience and literature search 

• Built framework to evaluate potential implementation ideas 

• Discussed both framework & implementation ideas with 
ESMD, SOMD, and SMD 

What follows: the set of Aerospace final presentation slides 
(slightly modified for length) followed by a description of the 
MD reactions) 






Selected Slides From AcrospscG 

Final Presentation 
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Introduction 


• Purpose 

- Develop and demonstrate a methodology for the comparative 
assessment of risks across the entire portfolio of NASA projects 
and assets 

• Assumptions 

- A “proof of concept”, demonstrating that disparate risks from 
across different parts of the agency can be compared 

- This presentation includes phase 1 only; Demonstration of 
framework and tool implementation is phase 2, which is TBD 

• What is Strategic Risk? 

- Working definitions: 

• “high level, long term risks, especially those that do not belong 
to a single program, mission area or Center” 

• “a risk for which the potential consequences are not fully 
comprehended and/or mitigated at the project, Directorate or 
Center level“ 

- The framework described here is not necessarily fixed to any 
particular definition or type of risk 




Executive Summary 


• Lessons learned from existing framework within NASA and 
Aerospace/SMC to capture strategic risks 

- BPR and management councils meetings present cross cutting strategic 
risks and top directorate risks with no clear ranking 

- Aerospace/SMC “Watch List” program utilizes Risk Manager to track risks 
outside the normal reporting chain 

• Prototype strategic risk framework developed 

- Proactive “checklist” approach to capture entire spectrum of risks 

- 5x5 rating (likelihood x consequence) modified to include tjmefmiTie 

- “Action” ranking for further study or need to involve NASA leadership 

- Risk Workbook and database 

• Range of implementation options to choose from 

- Active Risk Manager- requires new policies and buy-in from Centers 

- Passive Risk Advisor- collaborative approach, might better fit NASA’s 
culture 

- Risk Council - least effort required, but easily marginalized 
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Strategic Risk Identification 



• Internal Sources 

- Bottom-Up 

• Risks identified at every level of NASA 

• Risks eventually rise to Strategic level via NPR B000.4A 

- Top-Down 

• Centers 

• Directorates 

• Mission Support Offices 

• Administrator Staff Offices 

• External Sources 

- Government Agencies 

• Work with other government agencies to identify cross agency strategic risks 

- DoD, specially Air Force and NRO, is expected to have numerous common risks 

• Pull from work published by other govt, agencies (e.g. GAO) relating to NASA 

- Industry 

• Discuss with industry partners what strategic risks they see in horizon 

- Academia 

• Encourage academia to identify and formulate strategic risks 

- Media 

• Review journalism, blogs and watchdog materials for strategic risks 

- Significant challenge in dealing with poor "signal to noise ratio " 

- Science Fiction Writers and other innovative means 

• Risks and opportunities could potentially be identified from fiction writers 

- Similar to programs using writers to generate terrorist scenarios 

Details will vary depending on specific implementation plan 



Normalizing Strategic Risks 




Approach 


• Normalization & Ranking to compare disparate types of Strategic Risks 

- Global questions to answer for each risk 

• How serious a risk is this? 

• Allocate resources for further study of risk issue? 

• Bring risk issue to the attention of NASA Leadership? 

• Two approaches were considered 

- “Relative” 

• Multiple-choice assessment factors 

• Different assessment criteria^ Different consequence rating 

• Reference: A Practical R&D Project-Selection Scoring Tool, Anne DePiante Henriksen and 
Ann Jensen Traynor, IEEE Trans, on Engineering Management, Vol. 46, No. 2, May 1999 

- “Absolute” 

• Common scale for all risks (e.g M “expected dollar loss”) 

• Approach used in CARMA 

• “Relative” normalization and ranking approach was developed to create Ranking 
Scores 


Developed a quantitative method to score a risk’s 1) Relative Risk 
2) Value of Further Study and 3) Value of Raising Awareness 
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Calculation of Relative Risk Score 

Risk: Becomes evident in 2011 that commercial ISS resupply is not feasihk 


Scenario #1 - parameters 
assigned by a generalist 

- Likelihood, L = 3 (roughly even 
chance) 

- Consequence, C = 4 

- Timing, T = 3 (since scenario 
occurs 3 to 5 years in future) 

• Assume Y = 0.5 to discount 
risks in future 

Relative Risk Score 

- R = 3.22 


Scenario #2 - parameters assigned 
by specialists 

- Divided by NASA’s strategic goals 

- Consequence : 

• SGI (retire Shuttle) = 3 

• SG2 (ISS) = 5 

• SG3 (balanced program) = 2 

• SG4 (CEV ASAP) = 2 

• SG5 (commercial partnerships) = 4 

• SG6 (Lunar return & Mars) = 2 

- Assume likelihood and timing remain 
the same 

Relative Risk Score 

- SGI, R = 3.00 

- SG2, R = 3.41 

- SG3, R = 2.71 

- SG4, R = 2.71 

- SG5, R = 3.22 

- SG6,R = 2.71 

- Overall (equal weighting), R = 3.00 
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Calculation of Action Ranking Scores 



• Action rating criteria 

- U = 3 (issue pretty well understood) 

- M = 2 (mitigation is possible, at added cost) 

- A = 4 (NASA leadership involvement required to approve mitigation) 

• Action priorities for a single input, based on R = 3.22 (Y,Z= 0.5) 

- Value of Further Study = 3.34 

- Value of Raising Awareness = 3. 02 


• Top 5 Action priorities for multiple inputs (split by strategic goals, as in previous chart): 



Further study SG5 (Commercial partnerships) 3.34 

Further study 

Raise to attention of leadership SG2 (ISS) 3.11 

Further study SG4 (CEVASAP) 3 06 


Different risk scenarios are merged and sorted - highest priority actions 
will rise to the top of the list 
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Considerations for Risk Workbook Development 

Con Ops 

• Who will fill in information? 

- At what level of NASA organization ? 

• Will there be a dedicated individual/team in charge of normalization? 

• Is workbook a tool to: 

- Provide a “computer” ranking as a starting point for final ranking by humans 

• Individuals can play with data and see it ranked by different criteria 

- Break deadlocks when humans cannot agree on what should be elevated 

• Individuals agree to lower level inputs but then use tool to agree on final 
rankings 

• How many inputs for balance: 

- Enough information to create rankings 

- Not so much that no-one wants to take time to report a risk 


Risk Workbook dependent on ConOps 
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Workbook Inputs/Outputs 

Required inputs 

• Ranking Scores calculated by workbook: 

1. Relative Risk 

2. Value of Further Study 

3. Value of Raising Awareness 

• Minimum information required (13 total) 

- Status (1 Input): Toggle (New, Pending, Closed) - activates/deactivates risk 

- Descriptive Inputs (5 Inputs): Title, Description, Level, Category 1, Category 2 

- Ranking Parameters (6 Inputs): Timeframe (T), Likelihood (L), Consequence 
(C), Leadership Involvement Need (A), Mitigation Potential (M), Depth of Risk 
Understanding (U) 

- Point of Contact / Documentation (1 Input) 


The desired ranking criteria outputs drove minimum inputs needed 
As example, Consequence varied as single or multi-step process 
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Workbook Inputs/Outputs 



• Additional descriptive inputs could be included with each risk 

- Provides more information to decision makers 

- Aids in assigning ranking parameters 

• 7 optional descriptive inputs: 

- Possible mitigation options 

- Cost of action 

- Risk type (discrete vs. gradient, e.g.) 

- Action urgency (lead time) 

- Political dimensions 

- Public relations effects 

- Why is risk enterprise-level? 



Risk Workbook 

Results 
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Implementation Options 



• Worksheet tool created and implemented will depend on ConOps chosen 

• Each risk assessment ConOps can be described by four axes 

1. Degree of Specialization 

2. Ownership of Risk List 

3. Sources of Risks 

4. Hierarchy 

- Each proposed ConOps is described by where it falls on each of those axes 

• Three possible ConOps presented 

- ConOps #1: Active Risk Manager 

- ConOps #2: Passive Risk Advisor 

- ConOps #3: Risk Council 

• Many more ConOps possible 

- Any proposed ConOps can be modified to Leadership needs and desires 



Implementation Options 
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Generalist - knowledge 
of how risk(s) affects j 4- 
NASA as a whole 


(of those who control/fill-in risks) 


| 

I Specialist - knowledge 
| of how risk(s) affects 
| specific project 




Gatekeeper - a single 
“risk manager” owns 
and maintains the risk 14- 
list (and decides what 
j risks are added/deleted) j 


Ownership of Risk List 




risk list is a I 


I "Wiki” style 
I public database that 
1 anyone can add to 




Actively seek out risks 
blogs, GAO reports, etc. 


RTOWW iW*&S>Pl&WyL’ 


1 Risks compiled at lower 
w 1 levels and brought up 
j the chain to fill out the 
I master strategic risk list 

%P-SW> uK, 




Hierarchical - a single 
“risk manager” is 
responsible for the final 
ranking of each risk 


VWV'W'V'fcrwvv?"* 




| Parallel - final ranking 
> | of risks is voted on by 
multiple people/groups 
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Summary of 3 alternative implementation options 


ConOps #1 : Active Risk Manager 



ConOps #3: Risk Council 



ConOps #2: Passive Risk Advisor 
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ConOps #1 : Active Risk Manager 



ConOps Responsibilities 

ID — Risk Manager & NASA Entities 
(Directorates, Centers, MSO, ASO) 
Collect - Risk Manager 

(can be fed by NASA Entities) 

Evaluate — Risk Manager (can request help) 
Manage - Risk Manager 
Communicate - NASA Entities to Risk 
Manager; Risk Manager to NASA 
Leadership 





ConOps Description 

• Risk Manger (part of PA&E) 

- All strategic risks pass though RM 

• Keeps a historical record of all 
NASA Strategic Risks 

• Actively seeks risks from internal 
& external sources 

• Iterates with internal sources as 
needed to clarify risks 

- Solely responsible for assigning 
values to Ranking Parameters 

- Determines risks to: 

• Further study 

• Monitor or accept 

• Raise to Leadership (periodically 
or as needed) 

* NASA Entities 

- Feed internal risks by NASA entities 
& collects from external sources 
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RM Elevates 
Strategic Risks 
for Leadership 


as power to reques 






ConOps #2: Passive Risk Advisor 


ConOps Responsibilities 

ID — NASA Entities 
Collect - NASA Entities 
(for their organization only) 

Evaluate - NASA Entities 

(may request advise from Risk Advisor) 
Manage - NASA Entities 
(for their organization only) 
Communicate - NASA Entities to NASA 
Leadership 



ConOps Description 

• NASA Entities 

- Responsible for risks within their own 
organization only 

- May request advise from Risk 
Advisor if desired 

- NASA Entities determine risks to: 

• Further study 

• Monitor or accept 

• Raise to Leadership (using 
current councils 

* Risk Advisor (part of PA&E) 

- Collects risks from whatever sources 
are available but does not have 
authority to request them 

- Keeps a list of known risks and uses 
worksheet to create a ranking to use 
if services requested 

- Will provide advise to NASA Entities 
but does not decide what is 
presented to leadership 
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ConOps #2: Passive Risk Advisor 



Risk Advisor coilects risks from whatever information is already 
generated by NASA Entities. Advises NASA Entities when solicited. 


ConOps #3: Risk Council 


ConOps Responsibilities 

]D - Risk Council & NASA Entities 
(Directorates, Centers, MSO, ASO) 

Collect - Risk Council 
Evaluate - Risk Council Members 
Manage — Risk Council with Worksheet Admin 
Communicate — RC Members provide 
Ranking Parameter values; WA provides a 
ranked risk list to Leadership 



- - • 
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ConOps Description 

• Risk Council (reps from NASA Entities) 

- All strategic risks pass though RC 

• Actively seeks risks from internal 
& external sources 

• Iterates with internal sources as 
needed to clarify risks 

- Each member is responsible for 
assigning values of Ranking 
Parameters to every risk 

- Determines risks to: 

• Further study 

• Monitor or accept 

• Worksheet Administrator 

- Keeps a historical record of all NASA 
Strategic Risks 

- Collects Ranking Parameter values 
from each RC Member 


- Uses worksheet to automatically rank 
risks from RC Member inputs 


- Passes ranked risk to Leadership 




ConOps #3: Risk Council 



External Risks 
Collected by RC 






Internal Risks ! 
Pushed to RC ! 
Pulled by RC r 
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m mmmr $ Ranking 



y Parameters 
Provided by each 
it RC Member 


Unranked 
Risk List 
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Risk Administrator runs Worksheet for the Risk Council. Worksheet acts 
as a tool to consolidate inputs from RC Members and auto rank risks. /yft 
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Other Implementation Considerations 






• Budget authorities look for uncommitted funds 

- If risk suggests delay, then budget might be in play 

• May reveal internal conflict 

- The Aerospace Watch List sometimes disagrees with program office 
assessments, which is why it is private 

• May create disincentives 

- Risk of a schedule slip in a particular subsystem may encourage other managers 
to “bet on the come” that their parallel project will get schedule relief 

- If managers game schedules, what is true Critical Path? 

• May increase risk further 

- “Pre-emptive” responses by contractor to protect reputation or avoid legal action, 
for example 

• Mitigation options may be controversial 

- Workforce impacts might create morale problems, for example 

• Naysayers may blow risk out of proportion 

- Congress gets its information from the newspapers 
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Mission Directorate Responses 



MD Response: ESMD 


ESMD uses Continuous Risk Management 

There is a tremendous amount of work done to formally 
track -1400 risks in ESMD 

They added a spot in their framework for a HQ-level risk 
manager 

However, ESMD was of the opinion that attempting to 
concatenate any risks across mission directorates would 
be quite expensive, and using the methodology shown 
here might make it more so 

- Guess of about $3M/year to implement 



MO Response: SOMD 

• SOMD uses a more informal method to track risks 

• Each division has a risk manager who reports out at 
monthly meetings 

• Rankings are done at the AA level informally 

- AA attempts to empower AAAs 

• SOMD did not see a problem with participating in a 
Strategic Risk session as long as it was infrequent and 
not too much of a burden 

- Preferred quarterly meetings of a “Risk Panel” similar to the last 
implementation framework presented 
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MD Response: SMD 


• SMD uses a more informal method to track risks 

• Each division has a risk manager who reports out at 
monthly meetings 

• Rankings are done at the AA level informally 

- AA attempts to empower AAAs 

• SMD was resistant to the idea of any sort of risk panel 

- Didn’t think it would add value to do infrequently 

- Too much burden to do it frequently 

- Tracking too many risks at mission levels to make an effective 
meeting 

- Too difficult to concatenate, compare risks across missions, 
much less across directorates, even less across strategic- 
mission risks 

- Appears to be HQ micromanagement/mistrust of MDs, centers 
projects 



Associate Administrator Response 



• Presented to Associate Administrator C. Scholese on 
12 / 1/2009 

• Generally positively received 

• Requested limited implementation of framework in Early 
2010 

- Focusing on cross-cutting risks as identified by the monthly 
NASA Baseline Performance Review 
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Next Steps 


Implementing a limited set of risks for tracking purposes 

- Will try to capture largest 5-10 risks as presented at the BPR 

- Will attempt quantitative classification if possible 

Determining receiving organization 

- Will involve receiving org in trial run 

Effort required will be assessed 

- Will help calibrate continual use of technique 

- Will help size # of risks to be tracked 



